js-tiktoken
TypeScript icon, indicating that this package has built-in type declarations

1.0.16 • Public • Published

⏳ js-tiktoken

tiktoken is a BPE tokeniser for use with OpenAI's models. This is a pure JS port of the original tiktoken library.

Install the library from NPM:

npm install js-tiktoken

Usage

Basic usage follows, which includes all the OpenAI encoders and ranks:

import assert from "node:assert";
import { getEncoding, encodingForModel } from "js-tiktoken";

const enc = getEncoding("gpt2");
assert(enc.decode(enc.encode("hello world")) === "hello world");

Readme

Keywords

none

Package Sidebar

Install

npm i js-tiktoken

Weekly Downloads

778,045

Version

1.0.16

License

MIT

Unpacked Size

22.4 MB

Total Files

32

Issues

25

Pull Requests

1

Last publish

Collaborators

  • davidduong